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ABSTRACT 


For over 25 years, the United States Navy has been designing, developing 
and maintaining software for embedded computer systems. Throughout this 
generation of Naval aviation software development, no collective analysis of the 
successes and failures in software development had been accomplished. To 
accomplish this task, this thesis evaluated aircraft software data from the 
Department of the Navy against two metrics: 1) did the original software 
development schedule have to be changed, and 2) did the software released to 
the fleet contain any major defects? This research has revealed that only 
about half of the original software development schedules were sustained 
without a milestone change being made. Also, software that was released to 
the fleet had no major deficiencies three out of four times. To further specify 
this information, it has been refined into categories of software language, size 
of program and type of software program. The results of this study will be 
beneficial to aviation program managers, software developers and software 


maintenance technicians. 
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I. INTRODUCTION 


A. GENERAL 

The use of computers and application software has become 
a part of everyday life in today’s society. The same can be 
said for military organizations in all areas of specialization 
(i.e. administrative, training, maintenance, operations, etc). 
However, the area where the Department of Defense (DOD) spends 
the most money for computer software is in the area of 
embedded computers [Ref. 1]. The one section of 
embedded computers where the user requirements are the most 
stringent iS in weapons systems, especially aviation systems 
where decisions have to be made instantaneously with little or 
no room for error. The United States Navy has been designing, 
developing and implementing software for embedded computer 
avionics systems for over 25 years, but as the requirements of 
computer systems have become more demanding and aircraft have 
become more sophisticated the problems with software have 
increased (at least observably so). The difficulties of 
software projects meeting the original schedule have been 
noted throughout the computer industry and DOD is not immune 
from this problem. "... Air Force General Bernard Randolph, 
chief of Air Force Systems Command, characterized software as 


the Achilles’ heel of weapons development. "On software 


schedules, we’ve got a perfect record: We haven’t met one 
VGC race oa (Ref. 2] Additionally, this problem is 
magnified for DOD aviation programs because of their 
tremendous size, complicated specifications, large budgets and 
high public visibility. But, do all Navy aviation software 
programs have a problem with meeting their schedule or it is 
just the publicized ones who get the notoriety? And, if 
software programs have problems being on-time is it generally 
for the same cause or are there different reasons for the 
delays? This thesis will answer these questions and analyze 
the software factors which cause problems before and after a 


program is released to the Fleet. 


B. SCOPE 

In an attempt to discover why some software projects are 
successful and others are not, an analysis was conducted of 
the life cycle management of Naval Aviation mission critical 
software. This analysis of the life cycle management of 
mission critical software will be made comparing current and 
historical data on the software which operates the mission 


computers of several aviation platforms in the fleet. 


C. METHODOLOGY 
This thesis was developed using a four step approach. 
First, a general idea of research interest was determined and 


later was more narrowly focused to fit research capabilities. 


Second, a thorough literature review and data search was 
conducted to discover past efforts in this area, in an effort 
to develop a working database. Next, questionnaires, field 
trips and phone conversations were conducted to gain as much 
specific information on each project as possible. Finally, 
all data was collected and analyzed allowing conclusions and 


recommendations to be made. 


D. FOCUS 

This research was designed to collect as much information 
on embedded computer systems and their software in Navy 
aircraft as possible. Due to the many different types of 
computer systems in aircraft and the difficulty in 
accomplishing a valid comparison between systems only one type 
of computer system was chosen to collect data about. The 
system which almost all aircraft have in common is the main or 
mission computer. The software that runs these mission 
computers 1S an Operational Flight Program (OFP) or 
equivalent. The following Navy and Marine Corps aircraft use 


OFPs or the equivalent and are included in this research: 


A-6E AH-1W 
AV-8B EA-6B 
R-2e F-L4A 
HH-60H/J E=3C 
S=35/58 SH-2 


SH-3 SH-60B 


SH-60F 


E. ORGANIZATION 

Chapter II provides background information for this 
thesis. The applicable regulations and guidelines that Navy 
program managers must follow in developing, implementing and 
maintaining an embedded computer system are summarized. 
Additionally, a more thorough explanation of the principal 
document used in embedded software development is given. 

The process of data collection for this research is 
explained in Chapter III. The unique terms for software 
development are defined along with the metrics used to compare 
software projects. An explanation of the inquiry used to 
collect the data is given in an effort to show what 
information was required. 

Chapter IV contains an explanation of how the data was 
analyzed. The process of data comparison and analysis is 
shown along with the numerical results obtained from this 
process. 

The conclusions and recommendations are contained in 
Chapter V. A more in-depth explanation of the results 
obtained in Chapter IV is given, plus other noteworthy facts 
collected during this research. Additionally, a section 
outline with possible follow-on topics from this area of 


research is included. 


II. REGULATIONS FOR SOFTWARE DEVELOPMENT 


A. OVERVIEW 
All Navy program managers (PMs) who are in charge of major 
defense systems that contain embedded computer resources must 
observe set standards and guidelines for software development. 
Each regulation is written to integrate all phases of military 
software life cycle management and covers either overall 
policy or specific details for software development. To more 
fully understand, the purpose of each regulation and what 
information a PM or software developer can obtain from them, 
a short synopsis of these major standards and guidelines is 
provided. 
1. DOD Directive 5000.29 
Published in 1976, this directive establishes policy 
for the DOD in the management and control of the development, 
acquisition, deployment and support of computer resources in 
major defense systems. This directive requires embedded 
computer resources to undergo validation and risk analysis, 
configuration management, and life cycle planning. To oversee 
and coordinate the policies of this directive the Management 
Steering Committee for Embedded Computer Resources (MSC-ECR) 
was created. Besides improving the management of embedded 


computer resources in major defense systems, this committee 


works to ensure that new computer research, development, 
technology and policy are a part of normal defense system 
acquisition process. DOD Directive 5000.29 is the basis of 
all other Department of the Navy (DON) instructions on 
managing embedded computer resources. 
2. DOD-STD-2167A 

This DOD standard is the keystone regulation for the 
entire software development process with all other regulations 
providing either implementation policy or support for specific 
phases of life cycle management. This standard sets the 
requirements to be used during acquisition, development and 
support of mission critical software systems. These software 
life cycle requirements are not only mandatory for DOD 
agencies but also for the contractor. The major phases for 
the software development process that 2167A recommends will be 
explained in more detail below. 

3. DOD-STD-2168 

This DOD standard works in conjunction with DOD-STD- 
2167A to establish the requirements for a software quality 
program. To fulfill these requirements, a process must be 
implemented to effectively resolve software problems by 
evaluating software quality, documentation and related 
activities in a timely manner. The requirements of this 
standard are applicable to DOD agencies and contractors during 


the entire software life cycle from acquisition to support. 


A. DOD-STD-1679A 
This DOD standard is the predecessor to DOD-STD-2167 
and lists the original DOD requirements for mission critical 
software development. This software development procedure was 
written to allow for changing operational requirements, 
reduction of life cycle costs and the highest degree of 
software reliability and maintainability. Since this was the 
software development standard before September 1986, most 
Naval aviation software programs in operation today are 
covered under this standard. 
5. DOD-HDBK-287 
This DOD handbook was published to assist government 
agencies in tailoring DOD-STD-2167A for either a software 
development contract or a software support contract. The 
handbook provides key concepts of DOD-STD-2167A and the 
factors that should be considered when tailoring a software 
contract to this DOD standard. 
6. MIL-STD-480B 
This military standard sets forth the requirements and 
procedures for configuration control in the acquisition of 
software items. An important part of configuration control is 
the correct process of making changes to an already approved 
configuration item. The documents used for requesting these 
changes are known as Engineering Change Froposals (ECPs), 


deviations or waivers. The procedures, formats and rules for 


submitting these documents for changes are provided in MIL- 
STD-480B to standardize this process. 
7. MIL-STD-1521B 
This military standard provides the requirements to be 
followed when conducting technical reviews and audits of 
computer systems and software. This standard lists general 
and specific requirements that both the contracting agency and 
the contractor must accomplish during each phase of review or 
audit. Like DOD-STD-2167A, this standard shall be tailored to 
use only the applicable requirements for the computer resource 
being acquired. 
8. SECNAVINST 5200.32 
This Secretary of the Navy Instruction (SECNAVINST) 
sets DON policy for managing embedded computer resources 
including software. The overall policy of this instruction is 
to ensure that all levels of DON project and acquisition 
management give proper emphasis to life cycle and software 
management, risk and cost analysis, and stabilization of 
computer resource requirements. This instruction supplements 
the policies and procedures of DOD Directive 5000.29. 
9. OPNAVINST 5200.28 
This Chief of Naval Operations Instruction (OPNAVINST) 
establishes the CNO policy for life cycle management of 
Mission-Critical Computer Resources (MCCR) under the Research, 


Development and Acquisition (RDA) process. This policy covers 


all MCCR including software that are an integral part of or 
are used in support of weapons systems. The purpose of this 
instruction is to ensure that all MCCR that support weapons 
systems are integrated into the same life cycle management 
process as the weapons system. This life cycle management 
process begins from the very start of the acquisition process 
and continues through the post-deployment software support 
(EDS5) phase. 
10. NAVELEXINST 5200.23 

This instruction, which was originally promulgated by 
the Naval Electronic Systems Command (NAVELEX) and now is 
administered by the Commander, Space and Naval Warfare Systems 
Command (COMSPAWARSYSCOM), is the current U.S. Navy guide on 
computer software life cycle management. Information on 
software engineering and life cycle management of the software 
acquisition process is provided for use by program managers. 
This instruction also provides some of the factors that are 
common software problems, how current DOD policies were 
established to respond to these problems and why each phase of 
computer software life cycle management is important. 

11. TADSTANDS A through E 

These Tactical Digital Standards (TADSTANDS) A through 
E, which are administered by COMSFAWARSYSCOM for the Navy, 
establish the standards to be used during system development 


and life cycle support. Each TADSTAND sets the policy or 


requirements for the standardization of one of five areas: 
definitions for embedded computer resources (ECR), computer 
interface devices, programming and design languages, reserve 
computer capacity and requirements for mission-critical 


systems software acquisition, development and support. 


B. DOD-STD-1679A AND DOD-STD-2167A DIFFERENCES 

As the two main standards that are used to guide MCCR 
development, it is important to understand what changes, if 
any, were made between the first standard DOD-STD-1679 and 
it’s successor DOD-STD-2167. One of the basic differences 
between the two standards is that DOD-STD-2167A is more 
current with computer technology and refers to the components 
of software as Computer Software Configuration Items (CSCI) 
while DOD-STD-1679A uses older terminology and refers to 
software components aS programs, subprograms, modules and 
units. The only software programs that are subject to DOD- 
STD-2167 are those which have either issued a request for 
proposal for full scale engineering development (FSED) or 
entered FSED after September 1986. Although, the spirit and 
intent of both DOD-STD-1679A and -2167 are very similar, both 
standards approach software development differently. This can 
be seen when comparing the detailed requirements of DOD-STD- 
2167A (explained in the next section) and the detailed 


requirements of DOD-STD-1679A listed below: 
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1. Software Performance Requirements 
2. Software Design 

3. Programming Standards 

4. Programming Conventions 

5. Software Implementation 

6. Software Generation 

7. Software Operation 

8. Software Testing 

9. Software Quality Assurance 

10. Software Acceptance 

11. Software Configuration Management 


12. Software Management Planning 


C. DETAILED REQUIREMENTS OF DOD-STD-2167A 

Since DOD-STD-2167A (from now on referred to as 2167A) is 
the current MCCR software development standard for DOD, it 
will be explained in more detail than DOD-STD-1679A (from now 
on referred as to 1679A). Again, the spirit and intent of 
these two instructions is virtually the same -- development of 
the best quality software. 2167A establishes specific 
software development management requirements which must be 
followed by DOD contracting agencies and contractors. 
However, the standards can be tailored by the contracting 
agency if a requirement is non-applicable. The tailored set 


of requirements for each software program will be specified in 
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the contract agreed upon by the contractor and the contracting 


agency. All detailed requirements prescribed by 2167A 
contain elements of the general requirements: software 
development management, software engineering, formal 


qualification testing, software product evaluation and 
software configuration management. The standard set of 
detailed requirements are as follows: 
1. System requirements analysis/design 

This section of 2167A requires the software contractor 
to conduct a thorough analysis of system specifications for 
consistent and complete software requirements, and to optimize 
computer resource allocations (1.e. hardware, software and 
personnel). Also, the contractor shall support all system 
reviews as specified in the contract, plus evaluate and 
collate by specified criteria the proper preliminary 
documentation. 

2. Software requirements analysis 

The software contractor is required to conduct reviews 
of the software specifications by the standards set in MIL- 
STD-1521 (from now on referred to as 1521) and to establish 
the baseline for the Computer Software Configuration Items 
(CSCE)e All engineering, interface and qualification 
requirements for each CSCI shall be documented by the 
contractor, and evaluation of software and interface 


requirements must also be performed by criteria set in 2167A. 
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3. Preliminary design 
The Preliminary Design Review (PDR) of the software is 
to be conducted by the contractor according to the procedures 
set forth in 1521. The PDR ensures that the following items 
have been developed: a Software Design Document (SDD) which 
contains separate preliminary designs for each CSCI and 
requirement allocations; a Software Test Plan (STP) for formal 
qualification tests for each CSCI; and a preliminary Interface 
Design Document (IDD) which contains the preliminary design of 
interfaces external to each CSCI. 
4. Detailed design 
The Critical Design Review (CDR) of the software is to 
be conducted by the contractor according to the procedures set 
forth in 1521. The CDR is more specific than the PDR. The 
CDR verifies that the detailed design for each CSCI has been 
accomplished and documented in the SDD and IDD. The CDR also 
verifies that specific test cases have been described for each 
formal qualification test and documented in the Software Test 
Description (STD). 
5. Coding and CSU testing 
This section of 2167A requires the contractor to code 
and test each Computer Software Unit (CSU) to ensure specified 
requirements are meet. If changes are necessary to the CSU 


code, then revisions to the design, documentation and code 


ee 


will be made by the contractor along with any necessary 
retesting. 
6. CSC integration and testing 

After CSU coding and testing is complete, then the 
CSUs are assembled together into the correct Computer Software 
Component (CSC). The contractor must integrate and test each 
CSC to ensure specified requirements are meet. If changes are 
necessary, then revisions to the design, documentation and 
code will be made by the contractor along with any necessary 
retesting of CSUs or CSCs. A Test Readiness Review (TRR) 
will be conducted by procedures set forth in 1521 to ensure 
the CSC integration and testing is complete. 

7. CSCI testing 

This section of 2167A software development is where 
the Formal Qualification Testing (FQT) is conducted for each 
ese If changes are necessary due to FQT results, then 
revisions to the SDD, IDD and code will be made by the 
contractor along with any necessary retesting of applicable 
CSU, CSC or CSCI. The STD sets the procedures to be used for 
the FQT with the results being recorded in a Software Test 
Report (STR). The contractor may also support the Functional 
Configuration Audit (FCA) and Physical Configuration Audit 


(PCA) if conducted during this section. 
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8. System integration and testing 
The contractor will support all areas of system 
integration and testing and make any revisions’ to 


documentation and coding including retesting as necessary. 


Additionally, if FCA and PCA are conducted during this 


section, then the contractor will support it. 


This total process is graphically displayed in Figure 


1 which is reproduced from DOD-STD-2167A: 
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III. DATA COLLECTION 


A. DEFINITION OF TERMS 
The following words and terms are defined from DOD-STD- 
2167A, MIL-STD-1521B, MIL-STD-480B, or TADSTANDS A and 
D: [Ref. 3] 
1. Computer Resources 
"The totality of computer equipment, computer programs, 
computer data, associated documentation, personnel, and 
supplies." 
2. Computer Software Component (CSC) 
A distinct part of a computer software configuration item 
(CSCI). CSCs may be further decomposed into other CSCs 
and Computer Software Units (CSUs). 
3. Computer Software Configuration Item (CSCI) 
"A configuration item for computer software." 
4. Computer Software Unit (CSU) 
"An element specified in the design of a Computer Software 
Component (CSC) that is separately testable." 
5. Configuration Item (CI) 
"An aggregation of hardware, firmware, software, or any of 
its discrete portions, which satisfies an end use function and 
is designated for configuration management." 


6. Critical Design Review (CDR) 


This review shall be conducted for each configuration item 
when detail design is essentially complete. For CSCIs, 
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this review will focus on the determination of the 
acceptability of the detailed design, performance, and 
test characteristics of the design solution, and on the 
adequacy of the operation and support documents. 


7. Embedded Computer (EC) 


A digital computer or processor that is an integral 
component, from the design, procurement, and operations 
point of view, of any tactical digital system. Thass 
definition includes microcomputer, microprocessor, etc. 


8. Embedded Computer Resources (ECR) 


The totality of operational and support software/firmware; 
embedded computers; data storage and display devices; 
interface standards; programming languages; support 
facilities ashore; training facilities; training support 
personnel; and personnel whose primary specialized 
educational experience and/or training is directed toward 
operation or maintenance of embedded computers. 
Specifically included are programmable calculators 
(PROCALS) that are electrically interfaced to tactical 
digital systems. 


9. Formal Qualification Testing (FQT) 

"A process that allows the contracting agency to determine 
whether a configuration item complies with the allocated 
requirements for that item." 

10. Functional Configuration Audit (FCA) 


A formal audit to validate that the development of a 
configuration item has been completed satisfactorily and 
that the configuration item has achieved the performance 
and functional characteristics specified in the functional 
or allocated configuration identification. In addition, 


the completed operation and support documents shall be 
reviewed. 


11. Mission-Critical Computer Resources (MCCR) 


Computer resources acquired for use as integral parts of 
weapons; command and Contre |, communications; 
intelligence; and other tactical or strategic systems 
aboard ships, aircraft, and shore facilities and their 
support systems. The terms also includes all computer 
resources associated with specific program developmental 
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test and evaluation, operational test and evaluation, and 
post-deployment software support including weapon system 
trainer devices, automatic test equipment, land-based test 
sites, and system integration and test environments. 

12. Physical Configuration Audit (PCA) 

"A technical examination of a designated configuration 
item to verify that the configuration item ’As Built’ conforms 
to the technical documentation which defines the configuration 
item." 

13. Preliminary Design Review (PDR) 

"For CSCIs, this review will focus on: (1) the evaluation 
of the progress, consistency, and technical adequacy of the 
selected top-level design and test approach, (2) compatibility 
between software requirements and preliminary design, and (3) 
on the preliminary version of the operation and support 
documents." 

14. Problem Reports 

Also referred to as Software Trouble Reports (STRs) or 
Program Trouble Reports (PTRs) - 

Problems detected in the software or its documentation 
shall be classified by priority as follows: 

a. Priority One 

(Also referred to as an Emergency PTR) - A software problem 
that does one of the following: 

(1)Prevents the accomplishment of an operational or 
mission essential capability specified by baseline 
requirements 

(2)Prevents the ooperator’s accomplishment of an 
operational or mission essential capability. 

(3) Jeopardizes personnel safety. 

Ds 3PrTorilty = lwo 

A software problem that does one of the following: 


(l)Adversely affects the accomplishment of § an 
operational or mission essential capability specified by 
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baselined requirements so as to degrade performance and for 
which no alternative work-around solution is known. 
(2)Adversely affects the operator’s accomplishment of 
an operational or mission essential capability specified by 
baselined requirements so as to degrade performance and for 
which no alternative work-around solution is known. 
15. Test Readiness Review (TRR) 
A review conducted for each CSCI to determine whether the 
software test procedures are complete and to assure that 
the contractor is prepared for formal CSCI testing. 
Software test procedures are evaluated for compliance with 
software test plans and descriptions, and for adequacy in 
accomplishing test requirements. 
B. METRICS USED 
The success of any software project can only be obtained 
from the standards by which it is judged. For this analysis, 
the metrics used to define a successful software program were: 
ly. Did the program meet the initial planned delivery date 
with the specified software requirements? 
2. Was the program operationally successful (i.e. no priority 
one or two Problem Reports were issued after the software was 
released to the fleet)? 
Other factors which must be considered in this comparison 
analysis are program size, computer language the program is 
written in, and type of program (i.e. new, an upgrade to an 


existing program or a maintenance fix for problems reported in 


a previous program). 
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C. DATA SEARCH 

The collection of data in any research is difficult, 
especially if there is not a place acting as a central data 
repository. Another factor is the political sensitivity of 
the data to the responsible organization. These two factors 
are true for software data collection in both the public or 
private sectors and understandably so. In searching for data 
sources for this research, the few possible software databases 
which might contain viable information could not be used 
because of data confidentiality. Since no current databases 
could be used or found, an office in the Naval Aviation 
Command (NAVAIR) with connections to all aviation computer 
systems was discovered which could be a central point for data 
collection. However, because of the immense amount of 
background information that was needed on aircraft software, 
the best source of information was decided to be from the 
Software Support Ree rees (SSAs) in the field. The SSA is 
an organization whose purpose is to provide software 
maintenance and support for one specific aircraft type after 
the software contractor has completed contractual obligations 
and delivered the software. The SSA also monitors all phases 
of software development by the contractor and has the most 
complete records on aircraft software development of any Navy 
organization. Using all possible sources of information was 


important; therefore, the sources of data for this research 
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have come from offices in NAVAIR, the SSAs of the aircraft 


types used and technical support contractors. 


om INQUIRY BACKGROUND 
1. Purpose 
A major difficulty in collecting data from the SSAs is 
due to the fact that they are not in one central location, but 
are dispersed across the United States. In order to collect 
the research data necessary, an inquiry or small questionnaire 
was developed to collect the information required for this 
analysis. The inquiries were mailed or faxed to the 11 SSAs 
which participated in this study. Each inquiry was followed 
up with telephone conversations to alleviate any of the 
questions that either side may have had about the information 
being requested or the data being supplied. The inquiry data 
collection procedures were continually updated to ensure 
conciseness, clarity and completeness. A copy of the final 
revision of the inquiry used for data collection is provided 
in Appendix A. 
2. Information Collected 
As shown in Appendix A, the inquiry collected 


information on specific items about the software, as follows: 


ie Hype - of arrerart. 
2. Type of computer system. 


3. Total number of Lines of Code (LOC) in software program 
today. 
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4. Computer language that the software program was written 
Berry's 

oe A copy or list of the Software Life Cycle Schedule 
(sometimes referred to as "Milestone Charts")for each 
version of the software program, in order to gain a 
pictorial perspective of the history of each software 
program. 


ous For each software version, an explanation of why 
schedule changes, if any, (noted in 5 above) were made. 


7. Number of LOC which were new or changed from one version 
of the software program to the next. 


8. Type of software program each version was (e.g. Initial, 
Upgrade, Maintenance or Other). 


9. For each software version, reasons for Priority one or 
two Problem Reports, if any. 


ILO Space for additional comments on aviation software 
development, specific or general. 
3. Definition of Inquiry Terms 

Some of the terms in the software field are vague and 
not well defined. To help alleviate this situation, the 
following words and terms used in the inquiry are more fully 
defined. 

a. Lines of Code (LOC) - executable statements and 
data definitions are counted, but not comment statements and 
headers. 

b. Software Type - classification of the purpose of 
the software version released to the fleet. 

(1) Initial —- the original release of a software 
program for a new aircraft type or configuration, or for a 


major change in computer hardware configuration. 
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(2) Maintenance - the release of a software program 
to correct the problems (priority one or two PTRs) of a 
previously released software version. 

(3) Upgrade - the release of a software program to 
enhance the capabilities of or provide new features to a 
current working software version. Upgrades may also contain 
some corrections to minor software problems from previous 
releases. 

(4) Other - any software release which does not 
fall under the three classes above. 

ee Software Version - the nomenclature used to 

differentiate between a previous software program and any 
changes or updates made to that previous software program 


(e.g. program A will come before program A.1l or program B). 
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IV. DATA ANALYSIS 

Although the information collected in this research may 
have been specific as to personnel and organizations, the 
results of this study will only be of a generic nature. The 
categorizing of data into specific software factors was done 
to provide useful information on software development without 
drawing attention to certain software programs. This high 
level of confidentiality of source data was established in 


order to obtain the most accurate and candid information. 


A. DATA PARAMETERS 

Chapter III Section B discussed the metrics used to 
determine a successful software program, while Section D 
discussed the information that was collected in the inquiry. 
From the data collected, it has been reported that when a 
milestone changed for a software version, it was never 
accelerated but instead was always delayed. It has also been 
reported that once a milestone was delayed that the entire 
software development schedule was also delayed including the 
software delivery date. Therefore, if the inquiry data 
reported that a milestone change had occurred then the answer 
to metric standard one, about the initial planned delivery 
date being on time, was NO. Similarly, the metric standard 


about program operational success was YES if the program had 
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no priority one or two Program Trouble Reports (PTRs) and NO 
if there were priority one or two PTRs. These two metric 
standards, milestone changes and priority one or two PTRs, 
were compared against the total number of software versions in 
the study. To provide more precise information, the total 
number of software versions were changed into more specific 
technological categories of 1) software language, 2) program 
size, and 3) software type. Each of these categories was then 
subsequently divided into more explicit subcategories in order 
to further refine the results as follows: 

1) The software language category was divided into 
assembler and CMS-2, the two languages used for all programs 
tm this study. 

2) The program size category was separated into three 
subcategories of programs in the size ranges: 0 - 90,000, 
90,000 — 200,000, and 200,000 and above bytes. 

3) The software type category, as previously defined in 
Chapter III Section D, was subdivided into initial, upgrade, 
Maintenance and other. 

Also, the reasons why these milestones changed will be 


given along with percentage of occurrence. 


B. METHODS OF DATA ANALYSIS 
The analysis techniques that were determined to be 


appropriate for this study were the relative frequency of 
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class data or percentages and the Chi-square independence 
test. 
1. Frequency of Class Data Analysis Method 

To accomplish the first analysis technique of 
frequency of class data, the data on each success metric, 
milestone change and priority one or two PTRs, was divided 
into YES and NO responses for each metric. These four numbers 
were then divided by the total number of software versions in 
the study in order to get the overall percentage of software 
versions which were not delayed, delayed, operationally 
successful or not operationally successful. Each software 
version was then viewed from the more technical categories of: 
software language, program size or software type. These 
categories were further refined into their respective 
subcategories so as to make the data more specific. First, a 
percentage of each subcategory was calculated by adding up the 
total number of software versions per subcategory and dividing 
this number by the total number of software versions. Next, 
each subcategory was divided into YES/NO responses for each 
success metric, milestone change and priority one or two PTRs, 
and a percentage was calculated. This was accomplished by 
totaling up all the subcategory software versions that did and 
those that did not have milestone changes (those that had 
priority one or two PTRs and those that did not) and dividing 


this number by the total number of software versions in that 
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subcategory. Finally, a percentage was calculated for each 
different reason for why milestones changed. To accomplish 
this, the total number for each reason was divided by the 
total number of overall reasons. 

2. Chi-square Independence Test Analysis Method 

To accomplish the second analysis technique, each 
category (software language, program size and software type) 
was tested for independence with each of the two metrics 
(milestone change and priority one or two PTRs) using the chi- 
square independence test. Additionally, the two metrics were 
tested for dependency with each other. The chi-square 
independence method tests two events for statistical 
independence which is defined as: "... if the occurrence (or 
nonoccurrence) of one of the events does not affect the 
probability of the occurrence of the other event." 
[Ref. 4] The term independence will be used in place 
of statistical independence. The chi-square independence test 
requires that null and alternative hypothesis be stated. 

The null hypothesis is that each category was 
independent of each of the two metrics, and the alternative 
hypothesis was that each category was dependent of each of the 
two metrics. The chi-square procedure uses the observed 
values for each sub-category as shown in Tables 1-3, and a 
calculated expected value to derive the chi-square value for 


testing. The expected value is calculated by multiplying the 
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row total of software versions by the column total of software 
versions and dividing by the total number of software versions 
for each sub-category. The expected values must satisfy two 
assumptions: 1) all expected frequencies are at least one; and 
2) at most 20 percent of the expected frequencies are less 
than five. A level of significance must be determined for the 
test along with the degrees of freedom for each table. The 
degrees of freedom are calculated by subtracting one from the 
number of rows and multiplying this number by the number of 
columns minus one. A critical value for the chi-square value 
is found by using a chi-square distribution table with the 
input values of significance level and the degrees of freedom. 


The chi-square value (X’*) is calculated using the formula: 


(O-E) 2 
) Pines a 


Where O is the observed value and E is the expected value. 

If the chi-square value is less than the critical value then 
the null hypotheses is not rejected and the two items being 
tested are independent of each other. If the chi-square value 
is more than the critical value then the null hypotheses is 
rejected and the two items being tested have some form of 


dependency on each other. [Ref. 5] 
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C. DATA RESULTS 
1. Frequency of Class Data Results 

Of the 68 different software versions reviewed in this 
study, 32 of them had milestone changes and 19 of them had 
priority one or two PTRs written on them after fleet release. 
This means that 47.1 percent of these software versions were 
delayed from their initial scheduled delivery date and 27.9 
percent of them were not operationally successful. The number 
of software versions which were successful by both metrics 
(did not have a milestone change and had no priority one or 
two PTRs) was 31 or 45.6 percent. Further refinement of this 
information can be seen when it is viewed in the technical 
categories: software language, program size and software type. 

In the area of software language, 44 (64.7 percent) of 
these software versions were in assembly language and 24 (35.3 
percent) were in CMS-2. The number of assembly language 
programs which had milestone changes was 19 (43.2 percent), 
while six (13.6 percent) had priority one or two PTRs. A 
total of 22 (50.0 percent) assembly software versions passed 
both success metrics. Whereas, the number of CMS-2 language 
programs which had milestone changes was 13 (54.2 percent), 
while 13 (54.2 percent) also had priority one or two PTRs. A 
total of nine (37.5 percent) CMS-2 software versions passed 
both success metrics. The above data is organized in Table l 


below. 
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TABLE 1 
SOFTWARE LANGUAGE METRIC COMPARISON 













-— —- 
























Sub- Milestone PaLoOnLey Passed Both 
category Changes 1 or 2 PTRs Metrics 
No. / Pct / TECC: / "ECE 








: 13.6% | 86.4% 
CMS=2 Toe/ de ey ih ay, idles’: 9 / 15 7 
24 / 35.3% }} 54.2% | 45.8% 54.2% | 45.1% Sos 62.453 


The category of program size had 19 (27.9 percent) 





software versions in the range of 0 — 90,000 bytes (small), 39 
(57.4 percent) software versions in the range of 90,000 - 
200,000 bytes (medium) and ten (14.7 percent) software 
versions in the range of 200,000 bytes and above (large). Of 
the 19 small-size software versions, 12 (63.2 percent) had 
milestone changes, eight (42.1 percent) had priority one or 
two PTRs and six (31.6 percent) successfully passed both 
metrics. Of the 39 medium-size software versions, 15 (38.5 
percent) had milestone changes, seven (17.9 percent) had 
priority one or two PTRs written on them after fleet release 
and 21 (53.8 percent) successfully passed both metrics. 
Finally, on the ten large-size software versions, five (50.0 
percent) had milestone changes, four (40.0 percent) had 
priority one or two PTRs written on them and four (40.0 
percent) successfully passed both metrics. The above data is 


organized in Table 2 below. 
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TABLE 2 
PROGRAM SIZE METRIC COMPARISON 


| Sub- Milestone Priority Passed Both 
category * ally a 1 or <2 PTRs Metrics 
No. / Pct No. Bet No. ; No. rs Pct. 


9 90 A000 | uz 27 as 8 / iy 6 eS eS / 
ino 27.9% || 635.2% | 36.8% AZ No? . O* 31.6% | 68.4% 


0 7000: = SRO 4 24 / 7 / S27. Za / 9 2 / 
200,000 38.5% | 61.5% W296 1 O25. es 53.8% | 46.2% 
39 / 57.4% 


200,000 





In the category of software type, seven (10.4 percent) 
software versions were initial, 41 (61.2 percent) software 
versions were upgrade, 19 (28.4 percent) software versions 
were maintenance and one (1.5 percent) software version was 
classified as other. Of the seven initial software versions, 
all seven (100 percent) had milestone changes and priority one 
or two PTRs written on them, and therefore zero (0.0 percent) 
passed both success metrics. Of the 41 upgrade software 
versions, 21 (51.2 percent) had milestone changes, five (12.2 
percent) had priority one or two PTRs written on them after 
fleet release and 21 (51.2 percent) successfully passed both 
metrics. Of the 19 maintenance software versions, four (21.1 
percent) had milestone changes, six (31.6 percent) had 
priority one or two PTRs written on them and 11 (57.9 percent) 
passed both metrics successfully. Finally, the one software 


version which was classified as cther had zero (0.0 percent) 
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milestone changes, one (100 percent) priority one or two PTR 
written against it after fleet release and did not pass both 
success metrics. The above data 18S organized in Table 3 


below. 


TABLE 3 


Se ey 


Sub- Milestone Priority Passed Both 
category Changes 1 or 2 PTRs Metrics 
Pct Now, PCL. No. / Pct. 


/ 
ves | nwo | ves | wo | ves | wo | 
/ 
/ 


No. / Pct No. 
7 


e Yes Y 
/ 0 / /- OR 0 pay; 
100% 0.0% 100% 0.0% 0.0% | 100% 
Zin] 20 / SF / Ste, Zi) 20 / 
51.2% | 48.8% 12.2% | 87.8% 51.2% | 48.8% 
| 4 15 / 6 / 13 / 11 / 8 / = 
21.1% | 78.9% 31.6% | 68.4% 57.9% | 42.1% 


Ons Le By 0 / 0 / se) 
0.0% | 100% 100% 0.0% 0.0% |; 100% 


Of the 32 so0ftware versions which had milestone 





changes, the data sources reported 68 reasons why these 
milestones changed. Only one of the 12 reasons listed for a 
milestone change in the inquiry in Appendix A was not chosen 
aS a possible answer. The reason that did not cause a 
milestone change was hardware reliability. The reasons for 
milestone changes and their percentages of occurrence are 


listed in Table 4. 
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TABLE 4 
REASONS FOR MILESTONE CHANGES 


| Hardware Changes Om oe Software Changes 


ECe 
17 . 6% 
Changing User Ze.0% Software 4.4% 
Requirements Reliability 
Budgetary Pressure I Sis System Integration 10.3% 
Problems 


Inadequate TO", 3 Political Decision 10. 3% 
Integration Time 


Inadequate Design 1.5% Inadequate 1.5% 
Time Development time 


2. Chi-square Data Results 















r 









The first chi-square independence test compared 
software language against the two metrics (milestone change 
and priority one or two PTRs). The null hypothesis for both 
tests was that software language and milestone changes (or 
priority one or two PTRs) are independent. The alternative 
hypothesis for both tests was that software language and 
milestone changes (or priority one or two PTRs) are dependent. 
The significant level used was 0.01 and the number of degrees 
of freedom is one which gives a corresponding critical value 
of 6.635. The chi-square value for milestone changes is 0.8 
and for priority one or two PTRs is 12.7. Therefore, the null 
hypothesis for the milestone changes test is not rejected, but 
the null hypothesis for the priority one or two PTRs test is 


rejected and the alternative hypothesis is accepted. Tee 
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appears that software language is statistically independent of 
milestone changes but statistically dependent of priority one 
or two PTRs. The results of these two tests are shown below 
iTable 5. 


TABLE 5 
SOFTWARE EENGUASE INL SS Ea Es TEST RESULTS 


_— oe 





Sub-category || Milestone Priority 
Changes 1 or 2 PTRs 
Observed / Observed / 
Expected Expected 


corunn gota | 22 | 26 | 62 P| o_o 


The second chi-square independence test compared 


program size against the two metrics (milestone change and 
priority one or two PTRs). The null hypothesis for both tests 
was that program size and milestone changes (or priority one 
or two PTRs) are independent. The alternative hypothesis for 
both tests was that program size and milestone changes (or 
priority one or two PTRs) are dependent. The significant 
level used was 0.01 and the number of degrees of freedom is 
two which gives a corresponding critical value of 9.21 for 
each test. The chi-square value for milestone changes is 3.2 
and for priority one or two PTRs is 4.6. Therefore, the null 
hypothesis for both tests is not rejected, and it appears that 


program size is statistically independent of milestone changes 
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and priority one or two PTRs. The results of these two tests 
are shown below in Table 6. 


TABLE 6 


Milestone Prd Or ney 
Changes 1 or 2 PTRs 
Observed / Observed / 
Expected Expected 





The third chi-square independence test compared 
software type against the two metrics (milestone change and 
priority one or two PTRs). Since there was only one software 
version of software type other, this single value was deleted 
from this test. Therefore, these two tests will only have 67 
values instead of 68 as the previous four test have had. The 
null hypothesis for both tests was that software type and 
milestone changes (or priority one or two PTRs) are 
independent. The alternative hypothesis for both tests was 
that software type and milestone changes (or priority one or 
two PTRs) are dependent. The significant level used was 0.01 
and the number of degrees of freedom is two which gives a 


corresponding critical value of 9.21 for each test. The chi- 
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Square value for milestone changes is 13.3 and for priority 
one or two PTRs is 23.8. Therefore, the null hypothesis for 
both tests is rejected, and the alternative hypothesis is 
accepted. It appears that software type is statistically 
dependent of milestone changes and priority one or two PTRs. 
The results of these two tests are shown below in Table 7. It 
should be noted that the results for the chi-square 
independence test for software type and milestone changes may 
not be totally valid, since one of the chi-square test 
assumptions as stated in chapter IV Section B subsection 2 


above has been violated. 


TABLE 7 


Sub-category || Milestone Priority 
Changes 1 or 2 PTRs 
| Observed / Observed / 

Expected Expected 


(eo 





The fourth chi-square independence test compared the 
two metrics (milestone change and priority one or two PTRs) 
against each other. The null hypothesis was that milestone 


changes and priority one or two PTRs are independent. The 
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alternative hypothesis was that milestone changes and priority 
one or two PTRs are dependent. The significant level used was 
0.01 and the number of degrees of freedom is one which gives 
a corresponding critical value of 6.635 the test. The chi- 
square value for milestone changes is 7.5. Therefore, the 
null hypotheses for the test is rejected, and the alternative 
hypothesis is accepted. It appears that milestone changes are 
statistically dependent of priority one or two PTRs. The 


results of this test are shown below in Table 8. 


TABLE 8 
METRICS TNREEENDENCE TEST RESULTS 


Milestone 
Changes 


column Total in nea, Joa 
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V. CONCLUSION 
This research has produced many interesting results. From 
the raw collected data to conversations with software 
developers, these results express the facts and opinions of 
Naval aviation software developers with respect to their 
specific aircraft platform. This study has compiled these 
results to present an overall view of Naval aviation software 


development. 


A. EXPLANATION OF RESULTS 
1. Explanation of Software Language Results 

As shown in Table 1, the language (assembly or CMS-2) 
that a software version is written in only has a significant 
difference in the priority one or two PTRs metric. CMS-2 
software versions being released to the fleet have a four 
times greater percentage in having priority one or two PTRs as 
assembly language versions. 

The chi-square independence test results of Table 5 
have shown that software language and milestone changes are 
statistically independent, while software language and 
priority one or two PTRs are statistically dependent. 

2. Explanation of Program Size Results 
The results in Table < also show that the size of the 


software program had no significant affect on the success of 
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a software version against the metrics. However, medium-size 
(90,000 - 200,000) software versions on average do slightly 
better against the metrics than large-size (200,000 and above) 
software version which do slightly better than small-size (0- 
90,000) software versions. 

The results from the chi-square independence test of 
Table 6 verifies that program size is statistically 
independent of the occurrence milestone changes and priority 
one or two PTRs. 

3. Explanation of Software Type Results 

The most significant result of this study is in the 
category of software type. As shown in Table 3, ALL initial 
software versions had to change their original milestone 
schedule, and when finally released to the fleet, they had 
major problems that had to be reworked. 

Table 3 also shows that upgrade software versions are 
two times more likely to have a milestone change as 
maintenance versions, but less than half as likely to cause 
priority one or two PTRs to be generated after the software is 
released to the fleet. However, both maintenance and upgrade 
software versions are approximately equal in passing both 
metrics (no milestone changes and no priority one or two 
PTRSs} . 

The results of the chi-square independence tests of 


Table 7 show that type of software version affects the 
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occurrence of whether a software version is delayed or will 
have problems after fleet release. 
4. Explanation of Reasons for Milestone Change Results 

The reasons for milestone changes, as summarized in 
Table 4, show that software reliability, budgetary pressure, 
inadequate design time and inadequate development time are 
rarely a reason for changing a milestone since all four 
reasons together account for only 9.9 percent of the problems. 
The most prominent reason for milestones to change is because 
of changing user requirements, which accounted for 22 percent 
of the changes. The second most prominent reason for 
milestones to change is because of software changes, which 
accounted for 17.6 percent of the changes. Hardware changes, 
system integration problems, internal/external political 
decisions and miscellaneous other causes (usually dealing with 
documentation) were each the reason for milestone changes 10.3 
percent of the time. 

5. Explanation of Software Metrics Results 

The results of the chi-square independence tests of 
Table 8 have shown that whether a software version is delayed 
or not does not affect the probability that the software 
version will have major reported problems after it is released 


to the fleet. 
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B. SUMMARIZING THE RESULTS 

The most important success factor in defining a successful 
software version is the category of software type. This 
factor was further confirmed with the results of the chi- 
square independence test, since software type was the one 
category which had a statistical dependence of milestone 
changes and priority one or two PTRs. 

"Maintenance" types of software versions are the most 
successful (57.9 percent) in staying on original schedule and 
being trouble-free after release to the fleet. However, 
medium-size software versions, “upgrade” types of software 
versions and assembly language software versions are 50 
percent or better at passing both metrics. 

In contrast, "Initial" types of software versions were 
never able to maintain original schedule or be released to the 
fleet without major problems being discovered afterwards. All 
other subcategories (excluding the software type subcategory 
of other) are nearly equal in their percentage (a narrow range 
between 30 - 40 percent) of successfully passing both metrics. 
The software subcategories are ranked by their success at 
passing both metrics in Table 9 below. 

Further analysis of the results has shown that of the 15 
software versions which had milestone changes due to "changing 
user requirements" (the reason cited most often for a 
milestone change), 12 of these changes occurred in upgrade 


EV Pe SOLevare Versions. Thrs result ss net totally unexpected 
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Since upgrade software versions, in an effort to enhance fleet 


user capabilities as much as possible, try until the very last 


minute to add the latest requests for new system features. 


Maintenance versions on the other hand are usually more stable 


since they are trying to correct problems of a current 


software version, and few new functions are normally added. 


[Ref. 6] 


TABLE 9 


—=-= a —_— ee ———_ 





Maintenance Type Versions 5@ 


. 0% 
| Medium Size Programs 5@. 6% 
50.08 
50.08 
40.0% 
|cus-2 versions 40.08 
40.08 
| Other Type Versions | 0.0% 


Q 


RECOMMENDATIONS 


Subcategory Percentage that 


RANKING OF SOFTWARE SUBCATEGORIES BY PERCENTAGE 


Passed Both Metrics 


As previously discussed, the category of software type 


produced some very noteworthy results and showed possible 


areas where improvements could be made. 


This section notes 


some of the shortfalls discovered in this study and recommends 


some solutions. 
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Situation ONE 

The inability of the initial software versions to be 
produced without having to change their original milestone 
schedules. 

Recommendation —- For initial versions where software is 
being developed for the first time for a new aircraft 
configuration or where computer hardware is being added and/or 
changed for an existing aircraft system, more time is needed 
in the development process. Changes could possibly be made in 
the method used to calculate software development time 
schedules and allow for more development time for original 
versions of a software program. To some extent this extra 
time could be used to better define the system specifications 
or ensure integration problems are more thoroughly worked out. 
It would provide a more accurate implementation schedule. 
Further research to determine a more exact method for 
calculating software development time could more fully define 
a list of factors which cause schedule delays. 

Situation TWO 

The most Significant result was that even after changing 
their schedules these initial software versions had serious 
software problems which were not discovered until the software 
was in the fleet. For instance, a software version was 
released to the fleet after successfully passing testing, and 
under normal flight situations the computer would stop 


working. 
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Recommendation 

The best solution to this problem is to ensure 
specifications are thoroughly defined and that all areas of 
testing are well specified and properly accomplished. 

Situation THREE 

The low success rate of upgrade software versions (48.8 
percent) compared to maintenance software versions (78.9 
percent) in being able to maintain the original development 
schedule needs to be increased. 

Recommendation 


The suggested solution to revamp this problem is similar 


to situation one above. More time is needed in the 
development schedule. This in turn requires a more accurate 
method for estimating the upgrade schedule. Consideration 


should be given to solidifying software specifications when 
originally planned and not allowing any new changes or 
enhancements to be added to this baseline. New changes or 
enhancements would be handled in future updates. 

If an urgent change or enhancement is needed "NOW", this 
change should be made to the current working software version. 
If determined to be of a less urgent status, then add it to 
the follow-on version to the current software under 
development. Adding a late change or enhancement to an 


already baselined software version only causes problems in the 
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development schedule. The later such a change is made the 
more costly and time consuming the problem becomes. 
[Ref. 7] 

Situation FOUR 

As noted in Chapter IV Section C, on the average 45.6 
percent of the software versions passed both metrics (no 
milestone changes and no priority one or two PTRs). This 
average needs to be raised. An initial goal of at least 50 
percent should be made. In keeping with the DOD 
implementation of Total Quality Leadership (or continuous 
process improvement) efforts should continue to improve in 
this area. 

Recommendation 

Applying the recommendations of situations 1, 2 and 3 can 
help improve this percentage. Also, a more thorough study of 
this specific problem could generate procedures which would 
improve the entire software development process for DOD and 


save the government time and money. 


D. OVERVIEW OF THE DOD SOFTWARE DEVELOPMENT PROCESS 

The DOD standard for software development, DOD-STD-2167A, 
and the entire software development process are well 
established, especially considering today’s knowledge of this 
process. However, software development is neither a science 
nor a strict engineering discipline. It is more like an art, 


ana Se aiLerveult~to manage. [Ret. °8] 
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The DOD situation is further compounded because it 
endeavors to develop software for computers whose use is being 
continuously updated or completely changed. When the DOD is 
developing software for an embedded aircraft computer system, 
it is not like developing software for a personnel computer 
(PC). Most aircraft computers are real-time or near real-time 
systems. The software in these aircraft must respond to 
Mnhumerous inputs and produce several outputs instantaneously 
without failure. If the software ina PC fails, the user can 
restart the computer and try the problem again. If an 
aircraft is in a combat situation or needs the software for 
aircraft control, the crew may not have time to restart the 
computer and this may cost the government the lose of an 
aircraft and a crew. As a result, military software has an 
important requirement for minimal or zero software faults. 

For its part, the DON as a whole does a remarkable job of 
managing and producing software for embedded aircraft computer 
systems. In developing aircraft software, the DON must take 
into account a continuously changing world situation and the 
bureaucracy of the appropriation process for receiving 
funding. Additionally, all aircraft missions and capabilities 
are different, and each aircraft type must have software 
developed for it that will integrate correctly with unique 
hardware and avionic systems. However, in the process of 
collecting data for this study the following points were 


noted: 
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1. Relationship between software developer and technical 
and operational testers. 

- The software developers work hard to give the fleet 
user the best possible product with the newest technology and 
features as quickly as possible. 

- The testing agencies want to give the fleet a high 
quality product. They strive for zero defects in the software 
and work to ensure that the product is capable of doing the 
mission it was designed for. 

It would seem that both organizations, the software 
developers and the testing community, are working for the same 
thing, a successful product for the fleet. However at times, 
developers believe that not all technical and operational 
testing is required for every version of software; testers 
believe that any change to a software version should go 
through some if not all forms of testing. The developer’s 
opinion is that testing will delay a good software product 
from being delivered quickly to the fleet. The tester’s 
opinion is that if a bad software product is delivered to the 
fleet then the fleet is going to be less capable than before. 
A defective new software version is even further delayed. 

The data collected for this study show that there have 
been software products which have successfully passed the 
testing process without problems, ones in which problems were 
found and returned for corrections, and those that were sent 


out to the fleet with problems that were later discovered. 
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This area of "when" or "if" a software version should be 
tested is important enough to warrant a study of the situation 
to see if the current process should be revamped. However, 
the bottom line is that fleet user will be the one who decides 
if the product can be used effectively to get the mission 
accomplished because product environments change and what was 
useful yesterday may not be correct for the situation today. 
2. Incorporation of ADA. 

ADA is the High Order Language (HOL) used for computer 
programming that DOD had developed in an effort to standardize 
computer programming, logistics and support from several 
languages to one software language. For the Navy, OPNAVINST 
5200.28 has mandated that "Ada is required for new 
developments and shall be phased into use for existing systems 
at the next major upgrade." [Ref. 9] Congress, in the 
fiscal year 1991 budget, mandated ADA to be used [Ref. 10]. 

There are two potential problems the Navy has in 
incorporating ADA into existing systems. First, ADA is a 
relatively new software language and as such the programming 
experience level of ADA programmers is small. Second and 
perhaps most important, all the current software engineers 
that work to develop software for Navy aircraft and have many 
years of experience in developing aviation software, have 
little or no experience with ADA. This study has shown all 


aircraft computer programming for the programs reviewed is in 
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either assembler or CMS-2. The Navy needs to establish a plan 
which considers any or a combination of the following points: 

1) provide the experienced software engineers training 
in the ADA language. They are valuable resources having 
developed their aircraft software for many years. 


2) all newly hired programmers should be trained in 


3) be prepared to incur the additional learning curve 
transition to ADA which must be figured into the development 
schedule. 

A thorough study and analysis of this situation will 
provide valuable information and alternatives to make the 
optimum decision. 

S32 PIR reporting. 

The use of program trouble reports for reporting 
software or system problems by the fleet user may be lacking 
for the following reasons: 

a. the aircraft crews may find a way to work-around 
a problem or discrepancy, but in so doing are adapting 
themselves to the discrepancy situation rather than making 
sure the software or system is performing as_ specified. 
Because the aircraft crew has discovered a suitable work- 
around for the problem, a PTR may not be written, and the 
discrepancy is not reported. 

low all aircraft crews may not know how to report 


software or system problems, or do not believe it is important 
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to take the time to write down a discrepancy if a suitable 
work-around can be used instead. This situation of aircraft 
crews being uninformed about the importance of PTRs can lead 
to numerous software problems that are not reported and later 
may cause more serious problems. 

4. Software testing. 

In the collection of data for this research, software 
developers have expressed the importance of thorough 
validation and testing. The use of independent validation and 
verification (IV&V) and stress testing to discover major 
software problems is essential. Lack or only partial 
completion of these two forms of testing have been a major 
factor in software being released for technical and 
operational testing with priority one and two deficiencies. 

5. Software Integration. 

The integration of computer hardware with the software 
that will be operating on it is always a factor software 
developers take into consideration. However, some of the 
reasons for software delay and priority one or two PTRs are 
from integration problems with other aircraft systems. The 
source of these integration problems come from new or updated 
weapon systems (including weapon ballistics), auxiliary 
computers, or other avionics systems. The cause of these 
integration problems is normally that the change to the other 
aircraft system is considered by its developer to be so minute 


that this change should not affect any other system. This 
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unfortunately is not always the case. The important key is 
communication between developers is needed when a change is 


made to a system that integrates with a computer. 


E. POSSIBLE FOLLOW-ON TOPICS 

The area of mission critical computer software for Naval 
aeatwon “-S Cratical for the. future: Further in-depth 
research beyond this study can greatly assist with future 
decisions on Naval aviation software development. The 
following suggested research topics can provide valuable 
information for making these decisions. 

1. A thorough study of individual aircraft platform’s 
software. From when the initial software requirements were 
made with the software developer for the original aircraft 
through the software life cycle to the current version. 

2. A detailed study of the entire software maintenance 
process and documentation of what NAVAIR, the Software Support 
Activities (SSAs) and the testing agencies do to make changes 
in weapon system software. 

3. A more in-depth study of why initial software versions 
have problems maintaining original software development 
schedule and even when their schedules are updated major 
software problems still occur (i.e., priority one or two 
PTRs) . 

4. Research to determine why software version schedules 


fail to be met in excess of 50 percent. Also why these 
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programs are delivered with an inordinately high rate of major 
errors discovered after fleet release. 

5. An examination on the amount of testing that is 
Statistically required for a software version after it has 
left the SSA (i.e. does it need TechEval and/or OT&E). 

6. Analysis to determine what affect ANSI/MIL-STD-1815A 
(Reference Manual for the ADA Programming Language) and its 
implementation directives will have on the- software 
development process and the software developers (contractors 
and the SSAs) since most of the programmers know either 
assembler or CMS-2. 

7. Thorough study of the ability of the software 
developers to deliver software versions within budget. This 
study could be combined with the data from this thesis which 
would produce the more classic software study of the software 
developers ability to meet cost and schedule requirements and 
the difficulties in meeting these two criteria. 

8. Incorporation of the results of any of these research 
topics into a decision support system (DSS). A DSS could 
assist program managers or software developers in making 
decisions in many areas of the software development process. 
These systems would improve the entire Naval aviation software 


development process. 
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F. LESSONS LEARNED 

The following lessons learned were noted during the entire 
life cycle of this research: 

1. To gain an understanding and appreciation of the area 
of research, the researcher must be immersed into the research 
environment. This may entail asking the wrong or foolish 
questions, but later on this will enable the researcher to ask 
the right questions and collect the correct data. 

2. Questionnaires are adequate for data collection but 
face-to-face interviews are a faster way to collect data. 
Additionally, in-person interviews have the added advantage of 
allowing the researcher to get a better understanding of the 
data environment. 

3. For all forms of data collection, allow ample time for 
personnel to respond to research questions, but set a FIRM 
last day for acceptance of data collection and stick to it. 
Direct follow-up will ensure a higher response rate. 

4. The researcher should make the original research 
objective realistic yet flexible. This allows the researcher 
to modify the objective for contingencies such as needed data 
1s not easily accessible ina timely fashion or not available 


at al 1. 


G. FINAL THOUGHTS 
This study has only scratched the surface for evaluating 


the software development process of different types of 


be Be 


aircraft. To gain a more thorough understanding of the 
successes and failures of the Naval aviation software 
development process, further research that concentrates 
specifically on each aircraft type is needed. This in-depth 
research will help to correct problems each development 
process has, while allowing all other aircraft types to 
benefit from their successes. 

DOD-STD-2167A was established with the intention of 
allowing each aircraft program enough flexibility to develop 
mission critical software under any feasible development 
method, while still furnishing an architecture to work with. 
However, all phases of the software development method that is 
selected must be thoroughly implemented otherwise milestone 
delays and major deficiencies in fleet released software 
occur. Most problems in Naval aviation software development 
seem to occur when user requirements change after software 
development has commenced or when a development phase is not 
completely performed (e.g. incomplete stress testing). 

With decreasing budgets, Naval aviation software 
development must become as efficient as possible. This will 
require improvements in all areas of software development and 
the overall commitment of everyone involved in this process to 
a total and integrated team or mission concept. This task 
will be difficult, but with the implementation of Total 


Quality Leadership, it will not be impossible. 
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APPENDIX A 


INQUIRY FOR DATA COLLECTION 


1. What type of aircraft is the computer software used 
for? (circle all applicable platforms) 


A. A-6 EyeAv =o 

C. EA-6 De -B=Z 

BE. F-14 Baws 

Ge 53 oe ois: 

1S eee ae J. SH-60B 
Keo hd OO 


L. Others (please specify) 


2. What type of computer system is the software to be 
used for? (circle all applicable computer systems) 


A. AN/AYK-10 B. AN/AYK-14 
C. ASN-123 D.--ASN-15s0 
Bas CP—3SB Ieee ©) 2 re) OL 

Ga IT DY=43 


H. Others (please specify) 


oe 


ce What is the total number of lines of code in the 
software program today and what software languages is it 
written in? 


4. Please provide a copy of the Software Life Cycle 
Schedule (Milestone charts) for as much of the software 
program history as is available (i.e. from the initial 
software program to the current fleet release). If the above 
is not possible, please annotate when and what Milestones were 
revised for each version of software.) 
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5. Using the Milestones in number 4 above, please list 
what Milestones were changed and why they needed to be revised 
from the previous estimate. (Note: a Version number or 
software baseline designator are considered the same.) 


Possible Answers May Be: 
Hardware changes B. Software changes 
Changing user requirements D. Hardware reliability 
Software reliability F. Budgetary pressures 
System integration problems H. Inadequate 
integration time 


QRO Y 


I. Political decision J. Inadequate design 
time 
K. Inadequate development time L. Others (specify 
below) 
A. Version number Milestone 


Reason(s) for change (circle all appropriate answer(s)): 
A B C D E FE G H Af J K L 


Other(s) (please specify) 


B. Version number Milestone 
Reason(s) for change (circle all appropriate answer(s)): 
A B C D E EF G H ai J K L 


Other(s) (please specify) 


C. Version number Milestone 
Reason(s) for change (circle all appropriate answer(s)): 
A B Cc D E EF G H I J K L 


Other(s) (please specify) 


og 


D. Version number Milestone 
Reason(s) for change (circle all appropriate answer (s)): 
A B C D E F G H I J K L 


Other(s) (please specify) 


E. Version number Milestone 
Reason(s) for change (circle all appropriate answer(s)): 
A B C D EB F G H I J K L 


Other(s) (please specify) 


F. Version number Milestone 
Reason(s) for change (circle all appropriate answer(s)): 
A B C D E EF G H I J K L 


Other(s) (please specify) 


G. Version number Milestone 
Reason(s) for change (circle all appropriate answer(s)): 
A B Cc D EB Bs G H aE J K L 


Other(s) (please specify) 


H. Version number Milestone 
Reason(s) for change (circle all appropriate answer(s)): 
A B G D EB EF G H i = K L 


Other(s) (please specify) 
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6. For each version of software in number 4 above, what is 
the number of lines of newly written or changed source code 


for each version of software from previous baseline? 
A. 
5B. 
Ce 


iD 


le 


Version 
Version 
Version 
Version 
Version 
Version 
Version 
Version 
Version 


Version 


For each version of software in number 4 above, 


number 


number 


number 


number 


number 


number 


number 


number 


number 


number 


Type would you classify it as? 
Possible Answers are: 


A. Initial release 


C. Maintenance release to fix 
Prverity Teor 2 PIRs: 


A. 
Bi. 


Ce 


Version 
Version 
Version 
Version 
Version 
Version 
Version 
Version 
Version 


Version 


number 


number 


number 


number 


number 


number 


number 


number 


number 


number 
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Number 


Number 


Number 


Number 


Number 


Number 


Number 


Number 


Number 


Number 


of lines 
of lines 
of lines 
of lines 
of lines 
of lines 
of lines 
of lines 
of lines 


of lines 


what 


B. Major upgrade release 


D. Other (please specify) 


Type 
Type 
Type 
Type 
Type 
Type 
Lye 
iTvee 
Dype 


lype 


Oz Were Priority 1/Priority 2 Problem Reports or 
Emergency PTRs written against any of the software versions 
within 3 years of Fleet Issue? NO / YES (circle one) 


If YES, please elaborate on what problem the software program 
had and how the problem was fixed. 


A. Software version number 


Problem 


Solution 


B. Software version number 


Problem 


Solution 


C. Software version number 


Problem 


Solution 
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D. Software version number 


Problem 





Solution 


EB. Software version number 


Problem 


Solution 


F. Software version number 


Problem 


Solution 
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9. Please include any other comments which may be of use 
for this software analysis. 
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APPENDIX B 


LIST OF ACRONYMS 


Chief of Naval Operations Instruction OPNAVINST 
Commander, Space and Naval COMSPAWARSYSCOM 
Warfare Systems Command 

Computer Software Component Csc 
Computer Software Configuration Items CSE 
Computer Software Unit CSU 
Configuration Item CL 
Critical Design Review CDR 
Department of Defense DOD 
Department of the Navy DON 
Embedded Computer EC 
Embedded Computer Resources ECR 
Engineering Change Proposals ECPS 
Formal Qualification Testing FOT 
Full Scale Engineering Development FSED 
Functional Configuration Audit FCA 
High Order Language HOL 
Independent Validation and Verification IV&V 
Interface Design Document IDD 
Lines of Code LOC 
Management Steering Committee for MsC—BCR 
Embedded Computer Resources 

Mission-Critical Computer Resources McCCGr 
Naval Aviation Command NAVAIR 
Naval Electronic Systems Command NAVELEX 
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Operational Flight Program 
Personnel Computer 

Physical Configuration Audit 
Post-Deployment Software Support 
Preliminary Design Review 
Program Managers 

Program Trouble Reports 
Programmable Calculators 
Research, Development and Acquisition 
Secretary of the Navy Instruction 
Software Design Document 

Software Support Activities 
Software Test Description 
Software Test Plan 

Software Test Report 

Software Trouble Reports 

Tactical Digital Standards 


Test Readiness Review 
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OFP 


PC 


PCA 


PDSS 


PDR 


PMs 


PTRS 


PROCALS 


RDA 


SECNAVINST 


SDD 


SSAS 


STD 


STP 


STR 


STRs 


TADSTANDS 


TRR 


1G: 
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